LVS / IPVS / Keepalived Load Balancing HOWTO
TTK Ciar, ttk (at) archive.org
HOWTO document, 2004-12-02
 
 
This document very briefly discusses installing and configuring Keepalived and Linux Virtual Server (LVS) on Linux for the purpose of providing load balancing of a service across multiple hosts.
 
 
Table of Contents
 
   1. Introduction
   2. Installation
   3. Configuring Using Keepalive
   4. Configuring Using Just LVS Alone
   5. Making It Go
   6. Monitoring
   7. Caveats
   8. Things To Do
 
 
1. INTRODUCTION
 
 
Linux Virtual Server is a function of the Linux kernel which allows for the expedient forwarding of connections to multiple servers on a network.
 
 
Keepalived is a userspace daemon which wraps the LVS interface (ipvsadm) and provides for rapid and transparent failover of multiple virtual hosts (hosts forwarding connections to real hosts) and multiple real hosts (hosts accepting connections and providing services).
 
 
The purpose of using LVS is to divide processing load across multiple servers for best performance.  Keepalived makes this robust by co-ordinating the removal of servers from the LVS pool as they go down, and adding them back into the pool when they come back up.
 
 
 
Return To Table of Contents
 
2. INSTALLATION
 
 
This section is intentionally light, since the necessary packages come with adequate installation documentation.  The meat is in the configuration section, following this one.
 
 
IPVS is an optional feature of the Linux kernel.  Your kernel must be compiled with the IPVS options enabled.  Joerg has compiled correct kernels for each of our archivectures.  They are located in ops03:/0/nnssi/kernel/2.6.9/, and are bzImage-via, bzImage-athlon, and bzImage-p4.  Put the appropriate parameters into admin:/0/cluster/cluster.txt for each virtual servers.  Changing kernels is not necessary for real servers. (Note: Real servers must be able to restrict their ARP replies to addresses not bound to physical interfaces, which requires a kernel patch, but all kernels currently used at The Archive already have this patch, so this does not require additional configuration work.)
 
 
Additionally, ipvsadm must be installed.  On debian systems this is as simple as running 'apt-get install ipvsadm'.  Otherwise the v1.1.7 package must be obtained from www.linuxvirtualserver.org and installed.
 
 
Keepalived can be obtained from http://www.keepalived.org/.  Follow the instructions in its internal documentation to install.
 
 
It is strongly recommended that the correct kernel be installed on the virtual servers, and then ipvsadm installed, and then keepalived installed.  Changing versions of any package after any other package has been installed may give rise to interoperational problems.
 
 
 
Return To Table of Contents
 
3. CONFIGURATION WITH KEEPALIVED
 
 
Since configuration of load balancing is in some ways simpler with keepalived, it will be described first.  The following section will describe a standalone LVS configuration.
 
 
Before configuration begins, you will need the following resources:
 
  * Two or more virtual servers, which will forward connections to real servers (for instance: lvs01, lvs02).
     Choose one virtual server to be the "master".  The rest will be "backup" servers.
 
  * Two or more real servers, which will provide the services (for instance: www1, www4, www6).
     (In the example below, the IP addresses for www1, www4, and www6 are 207.241.228.105, 207.241.228.168, and 207.241.229.86 respectively.)
 
  * One or more "floating" IP addresses which have not been assigned to any host.
     (In the example below, 207.241.224.241)
 
 
Configuration of virtual servers: For each virtual server, the following lines must be entered into its /specials/rc.final file
(located at admin:/0/cluster/Specials/):
sysctl -w net.ipv4.ip_forward=1
/usr/sbin/keepalived -f /specials/keepalived.conf
 
Appropriate keepalived.conf files must then be made for each virtual server.  They should all be identical, except that the master server should specify master status.  Brief descriptions of some parameters are described.  For more complete descriptions, please view the Keepalived Users Guide at http://www.keepalived.org/pdf/UserGuide.pdf).
global_defs {
   notification_email {
     yourname@archive.org
   }
   notification_email_from yourname@archive.org
   smtp_server 207.241.230.143  # <-- Replace this with real mail-dev.archive.org IP address
   smtp_connect_timeout 30
   lvs_id LVS_FOO  # <-- Replace this with a unique name (like LVS_WB or LVS_CGI)
}

# The vrrp_sync_group defines the group of hosts amongst which the floating IP address 
# should be shared.  This one is very simple, but VRRP can be used to define very 
# complicated failover organizations, involving multiple services, with different 
# virtual host groups for each service.
vrrp_sync_group VGX { # <-- Replace VGX with a unique name
    group {
      VI_Z  # <-- replace VI_Z with a unique name
    }
}

# The vrrp_instance describes the VRRP parameters for this virtual server.
vrrp_instance VI_Z {
    state MASTER  # <-- make this BACKUP for backup virtual servers
    interface eth0
    virtual_router_id 42  # <-- replace 42 with a unique number
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1414  # <-- replace 1414 with a unique number
    }
    virtual_ipaddress {
        207.241.224.241/21 # vip -- "floating" ip of LVS virtual host pool
    }
}

# Here we describe the real servers being multiplexed off a given floating ip.
virtual_server 207.241.224.241 80 { # 80 = http service, for www load balancing
    delay_loop 10  # number of seconds between real server connectvity checks
    lb_algo rr     # use round-robin connection allocation algorithm
    lb_kind DR     # use direct routing; can also be NAT, but don't do that.
    persistence_timeout 9600  # Joerg says to use three hour timeouts, and I trust him.
    protocol TCP   # allocate connections to TCP port 80, as opposed to UDP port 80.

    real_server 207.241.228.105 80 { # www1
        weight 1     # higher number == give this machine more connections.
        TCP_CHECK {  # use simple TCP connectivity test for detecting up/down status.
          connect_timeout 3   # server is down if connection takes > 3 seconds
          connect_port    80  # test connection to TCP port 80
        }
    }
    real_server 207.241.228.168 80 { # www4
        weight 1
        TCP_CHECK {
          connect_timeout 3
          connect_port    80
        }
    }
    real_server 207.241.229.86 80 { # www6
        weight 1
        TCP_CHECK {
          connect_timeout 3
          connect_port    80
        }
    }
}

 
Configuration of real servers: For each real server, the following lines must be entered into its /specials/rc.final file
(located at admin:/0/cluster/Specials/):
ip route add 224.0.0.0/4 dev eth0
sysctl -w net.ipv4.conf.lo.hidden=1
sysctl -w net.ipv4.conf.all.hidden=1
ip addr add 207.241.224.241/32 dev lo
 
 
Return To Table of Contents
 
4. CONFIGURATION OF STANDALONE LVS
 
 
If you do not care about failover, or keepalived is giving you grief, LVS can be used by itself.
 
 
Before configuration begins, you will need the following resources:
 
  * One virtual IP address (vip) which has not been allocated to any machine.
     (In the example below, 207.241.224.240).
 
  * One virtual server, which will forward connections to real servers.
 
  * Two or more real servers, which will provide the services (for instance: www1, www4, www6).
     (In the example below, the IP addresses for www1, www4, and www6 are 207.241.228.105, 207.241.228.168, and 207.241.229.86 respectively.)
 
 
Make sure an LVS-enabled kernel and the ipvsadm utility has been installed on the virtual server (see INSTALLATION section).
 
 
The following lines must be entered into the virtual server's /specials/rc.final file
(located at admin:/0/cluster/Specials/):
sysctl -w net.ipv4.ip_forward=1
ip addr add 207.241.224.240/21 dev eth0
ipvsadm -D -t 207.241.224.240:80
 
Futhermore, one line must be entered into the virtual server's /specials/rc.final file for every real server:
ipvsadm -a -t 207.241.224.240:80 -r 207.241.228.105 -g
ipvsadm -a -t 207.241.224.240:80 -r 207.241.228.168 -g
ipvsadm -a -t 207.241.224.240:80 -r 207.241.229.86  -g
 
Configuration of real servers: For each real server, the following lines must be entered into its /specials/rc.final file
(located at admin:/0/cluster/Specials/):
sysctl -w net.ipv4.conf.lo.hidden=1
sysctl -w net.ipv4.conf.all.hidden=1
ip addr add 207.241.224.240/32 dev lo
 
 
Return To Table of Contents
 
5. MAKING IT GO
 
 
To make it all go, either reboot the real and virtual servers (in that order), or type the commands into the shell, which you added to each machine's rc.final.
 
 
There is also a script called lvstool, probably located a homeserver:/home/bill/bin/lvstool, which makes remote standalone LVS initialization easy.  It should be run as root from admin.  Maybe someday there will be an equivalent tool for using keepalived.  Bug John Berry about allocating some software engineer's time to it.
 
 
 
Return To Table of Contents
 
6. MONITORING
 
 
One line should be added to cluster.txt for every VIP (floating IP), which will cause Nagios to monitor the VIP like ordinary servers:
www 00:00:00:00:00:00 LB WWW-VIP www 207.241.224.241 -
This will cause Nagios to perform PING and SSH checks, which will be done on whichever Virtual Server is currently holding the VIP.  In this case (www), an apache check will also be performed on port 80, which will go through the VIP normally.
 
The easiest way to monitor the state of the loadbalancer is with ipvsadm.  Run it on a virtual server with the '-l' option to see the current state of LVS.
 
 
List current stats:
lvs01:~# ipvsadm -l
IP Virtual Server version 1.2.0 (size=16384)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  iahost-241-224-241.archive.o rr persistent 9600
  -> www6.archive.org:www         Route   1      38         1668      
  -> www4.archive.org:www         Route   1      33         1400      
  -> www1.archive.org:www         Route   1      37         1874      
Other handy variants:
lvs01:~# ipvsadm -l -n
lvs01:~# ipvsadm -l -n --rate
lvs01:~# ipvsadm -l -n --stats
lvs01:~# ipvsadm -l -c
Particularly recommended:
lvs01:~# watch ipvsadm -l -n --rate
See "ipvsadm --help" for full options.
 
 
If you are using keepalived, run "watch ipvsadm -l -n --rate" on each virtual server so you can see the connection forwarding "jump over" to a backup server when the master goes down.
 
 
 
Return To Table of Contents
 
7. CAVEATS
 
 
When Direct Routing (DR) mode is used, LVS only works correctly when the virtual servers and real servers are all on the same local network.  It is possible to do more complex configurations using TUN and NAT, but this is more complex and outside the scope of this HOWTO.  See the Keepalived Users Guide for more information.
 
 
Whenever the IP of a real server changes, this change must be reflected in each keepalived.conf file, and then keepalived restarted.  This can be done by using "ps aux | grep keepalived" to obtain the pid of keepalived (it will be the first process of the three that show up), "kill <pid>", and then "keepalived -f /specials/keepalived.conf".  However, simply rebooting the server on any configuration change might be preferable.  Users of keepalived have reported problems with the kill + re-run process, and there should be no interruption of services when a single server is rebooted anyway.
 
 
The same goes for when the IP of your mail server changes.
 
 
 
Return To Table of Contents
 
8. THINGS TO DO
 
 
  * Enhance lvstool with monitoring function.
 
  * Write analog to lvstool for keepalived.
 
  * Update this HOWTO with directord information.
 
  * Discuss issues with flapping servers.
 
  * Make up a Nagios check for monitoring.
 
Return To Table of Contents